Development and testing of a multi-lingual Natural Language Processing-based deep learning system in 10 languages for COVID-19 pandemic crisis: A multi-center study

Purpose The COVID-19 pandemic has drastically disrupted global healthcare systems. With the higher demand for healthcare and misinformation related to COVID-19, there is a need to explore alternative models to improve communication. Artificial Intelligence (AI) and Natural Language Processing (NLP) have emerged as promising solutions to improve healthcare delivery. Chatbots could fill a pivotal role in the dissemination and easy accessibility of accurate information in a pandemic. In this study, we developed a multi-lingual NLP-based AI chatbot, DR-COVID, which responds accurately to open-ended, COVID-19 related questions. This was used to facilitate pandemic education and healthcare delivery. Methods First, we developed DR-COVID with an ensemble NLP model on the Telegram platform (https://t.me/drcovid_nlp_chatbot). Second, we evaluated various performance metrics. Third, we evaluated multi-lingual text-to-text translation to Chinese, Malay, Tamil, Filipino, Thai, Japanese, French, Spanish, and Portuguese. We utilized 2,728 training questions and 821 test questions in English. Primary outcome measurements were (A) overall and top 3 accuracies;(B) Area Under the Curve (AUC), precision, recall, and F1 score. Overall accuracy referred to a correct response for the top answer, whereas top 3 accuracy referred to an appropriate response for any one answer amongst the top 3 answers. AUC and its relevant matrices were obtained from the Receiver Operation Characteristics (ROC) curve. Secondary outcomes were (A) multi-lingual accuracy;(B) comparison to enterprise-grade chatbot systems. The sharing of training and testing datasets on an open-source platform will also contribute to existing data. Results Our NLP model, utilizing the ensemble architecture, achieved overall and top 3 accuracies of 0.838 [95% confidence interval (CI): 0.826–0.851] and 0.922 [95% CI: 0.913–0.932] respectively. For overall and top 3 results, AUC scores of 0.917 [95% CI: 0.911–0.925] and 0.960 [95% CI: 0.955–0.964] were achieved respectively. We achieved multi-linguicism with nine non-English languages, with Portuguese performing the best overall at 0.900. Lastly, DR-COVID generated answers more accurately and quickly than other chatbots, within 1.12–2.15 s across three devices tested. Conclusion DR-COVID is a clinically effective NLP-based conversational AI chatbot, and a promising solution for healthcare delivery in the pandemic era.

Development and testing of a multi-lingual Natural Language Processing-based deep learning system in 10 languages for COVID-19 pandemic crisis: A multi-center study.

Purpose: The COVID-19 pandemic has drastically disrupted global healthcare systems. With the higher demand for healthcare and misinformation related to COVID-19, there is a need to explore alternative models to improve communication. Artificial Intelligence (AI) and Natural Language Processing (NLP) have emerged as promising solutions to improve healthcare delivery. Chatbots could fill a pivotal role in the dissemination and easy accessibility of accurate information in a pandemic. In this study, we developed a multi-lingual NLP-based AI chatbot, DR-COVID, which responds accurately to open-ended, COVID-19 related questions. This was used to facilitate pandemic education and healthcare delivery. Methods: First, we developed DR-COVID with an ensemble NLP model on the Telegram platform (https://t.me/drcovid_nlp_chatbot). Second, we evaluated various performance metrics. Third, we evaluated multi-lingual text-to-text translation to Chinese, Malay, Tamil, Filipino, Thai, Japanese, French, Spanish, and Portuguese. We utilized 2,728 training questions and 821 test questions in English. Primary outcome measurements were (A) overall and top 3 accuracies; (B) Area Under the Curve (AUC), precision, recall, and F1 score. Overall accuracy referred to a correct response for the top answer, whereas top 3 accuracy referred to an appropriate response for any one answer amongst the top 3 answers. AUC and its relevant matrices were obtained from the Receiver Operation Characteristics (ROC) curve. Secondary outcomes were (A) multi-lingual accuracy; (B) comparison to enterprise-grade chatbot systems. The sharing of training and testing datasets on an open-source platform will also contribute to existing data. Results: Our NLP model, utilizing the ensemble architecture, achieved overall and top 3 accuracies of 0.838 [95% confidence interval (CI): 0.826-0.851] and 0.922 [95% CI: 0.913-0.932] respectively. For overall and top 3 results, AUC scores of 0.917 [95% CI: 0.911-0.925] and 0.960 [95% CI: 0.955-0.964] were achieved respectively. We achieved multi-linguicism with nine non-English languages, with Portuguese performing the best overall at 0.900. Lastly, DR-COVID generated answers more accurately and quickly than other chatbots, within 1.12-2.15 s across three devices tested. Conclusion: DR-COVID is a clinically effective NLP-based conversational AI chatbot, and a promising solution for healthcare delivery in the pandemic era.

Perception survey of crisis and emergency risk communication in an acute hospital in the management of COVID-19 pandemic in Singapore.

BACKGROUND: Emergency risk communication is a critical component in emergency planning and response. It has been recognised as significant for planning for and responding to public health emergencies. While there is a growing body of guidelines and frameworks on emergency risk communication, it remains a relatively new field. There has also been limited attention on how emergency risk communication is being performed in public health organisations, such as acute hospitals, and what the associated challenges are. This article seeks to examine the perception of crisis and emergency risk communication in an acute hospital in response to COVID-19 pandemic in Singapore and to identify its associated enablers and barriers. METHODS: A 13-item Crisis and Emergency Risk Communication (CERC) Survey, based on the US Centers for Disease and Control (CDC) CERC framework, was developed and administered to hospital staff during February 24-28, 2020. The survey also included an open-ended question to solicit feedback on areas of CERC in need of improvement. Chi-square test was used for analysis of survey data. Thematic analysis was performed on qualitative feedback. RESULTS: Of the 1154 participants who responded to the survey, most (94.1%) reported that regular hospital updates on COVID-19 were understandable and actionable. Many (92.5%) stated that accurate, concise and timely information helped to keep them safe. A majority (92.3%) of them were clear about the hospital's response to the COVID-19 situation, and 79.4% of the respondents reported that the hospital had been able to understand their challenges and address their concerns. Sociodemographic characteristics, such as occupation, age, marital status, work experience, gender, and staff's primary work location influenced the responses to hospital CERC. Local leaders within the hospital would need support to better communicate and translate hospital updates in response to COVID-19 to actionable plans for their staff. Better communication in executing resource utilization plans, expressing more empathy and care for their staff, and enhancing communication channels, such as through the use of secure text messaging rather than emails would be important. CONCLUSION: CERC is relevant and important in the hospital setting to managing COVID-19 and should be considered concurrently with hospital emergency response domains.